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ABSTRACT 

This is a final report of support received through 
the National Institute of Education (NIE) to assist in research on 
the writing ability of Unitod States children. The study began under 
the auspices of the International Association for the Evaluation of 
Educational AchievemeAt (lEA) in order to contribute to a cross 
national assessment of students' writing proficiency. Aims of the 
study were: (1) to provide a conceptualization of the domain cf 
writing; (2) to explore and describe various aspects of teaching 
written composition; (3) to describe how students in different 
countries respond to assignments; and (4) to sst hypotheses about 
factors assumed to affect performance in wri-cten composition. A model 
was developed for the study which included three populations 
(representing three levels of schooling) and a series of instruments 
to be used. Research was organized arcund: (1) sampling of 
participant schools; (2) describing replacement strategy and 
procedure; and (3) describing the procedure for selecting classes. 
The remainder of the research addressed aspects of the sampling plan 
and procedure which were unique to the three individual populations 
of the model. Results of preliminary data analysis are presented in 
tables. A supporting document. Task 1 Training and Scoring Materials, 
is provided in ar. appendix. (LMO) 
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Introduction 

This constitutes a final report of the support received through the 
National Institute of Education (through November, 1985) to assist in 
research on the writing ability of United States children. The study, 
begun under the auspices of the International Association for the 
Evaluation of Educational Achievement (lEA)^ is to contribute to a cross 
national assessment of students' writing proficiency. The research base 
for this effort derived from earlier CSE activities supported through NIE 
funds (from 1976-1981), and the design support for this endeavor was 
provided by the Spencer Foundation. 

Considerable difficulty was experienced in trying to consolidate 
research support to permit this study to go forward. The MacArthur 
Foundation provided critical support to permit the data collection and to 
defray some costs of scoring. The NIE support assisted us in training 
raters, scoring, and initial data cleaning end analysis costs. Therefore, 
this report describes efforts supported by the NIE, but is not a final 
report of the study itself. We are in continued negotiations to secure 
funds for the full data analysis of this study. 



This report will describe the background of the study briefly, but its 
fo'^us will be on the training and scoring of the student essays. Prelim- 
inary data analyses will be reported, but we expect these to be rechecked, 
and to conduct additional analyses, before we forward them on to the lEA or 
publish a report on the U.S. National Study. 
Background 

Written composition skills is one of the clear school-based learning 
tasks on which there is almost universal agreement. Yet, the quality of 
students' writing in the USA is continually judged poor, whether the 
feedback comes from college admissions testing or the world of employment. 
Testimony to the difficulties in teaching writing can be inferred by the 
rapid expansion of remedial programs offered at institutions of higher 
learning to bring entering students' writing skills up to acceptable 
levels. And college-bound students are thought to perform at the highest 
levels in writing of all American secondary school students. 

State departments of education have responded, as has the National 
Assessment of Educational Progress (NAEP), with measures designed to assess 
writing. Until recently, these measures were often "indirect", that is, 
they were made up of multiple choice items which correlated at moderate 
levels with actual writing performance on essays. In the course of the 
minimum competency testing movement, NAEP, a majority of states, and rrany 
local scnool districts have developed their own af^sessments of student 
composition using actual paragraphs or themes prepared by students. 

While much of this effort is commendable, a good deal of it is 
flawed. First, the minimalist focus in some testing programs results in 



students writing on tasks or topics where bare convention rather than 
thinking and organizing is the major outcome. For instance, many districts 
and states have at one time used a "letter" task and have emphasized 
letter format, e.g., location of the salutation, rather than letter 
content. Furthermore, many tasks are artificial, have no well developed 
audience (other than the teacher) or ask students to write about topics 
about which they may be drastically uninformed. 

On psychometric grounds, many of these writing assessments are also 
inadequate. Because most assessment was developed for the purpose of 
individual placement, e.g., who to exempt from remedial Englislj, underlying 
psychometric models focussed on rankings of students. Now, however, the 
explicit purposes of assessment have shifted to improving individual and 
group performance, rendering these normative models inappropriate. In 
fact, new psychometric models and validation strategies are available, in 
part developed with resources of the National Institute of Education. They 
are being applied in a subset of states, but should be more widely 
distributed. 

Another serious issue in writing assess;:.ont is scoring or rating 
strategies. Because of the importance of improving student performan^., 
the strategy recommended in this study, and being implemented in the larger 
international plan, provides diagnostic information. This strategy 
features an analytic scoring rubric, a guide that provides sub-scores on 
writing features. Analytic scoring has in the past emphasized only 
sentence-level flaws, e.g., spelling, but the guides proposed herein employ 
subscales designed to assess maturity and depth of ideas, sentence. 
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paragraph and theme organization, and style and audience, in addition to 
skills such as punctuation, syntactic appropriateness, and spelling. From 
suc^ efined information, teachers and other instructional program planners 
can determine what aspects of instruction need modification and develop 
targeted rather than more costly global revisions of curricula. 

An additional concern present in the conduct of most written 
composition assessment efforts is the cost of scoring. An initial 
investment was made wherein the composition raters are trained to a high 
level of agreement (and there is amp""e evidence such agreement is 
possible). The use of these scoring procerlures reduced the amount of time 
spent on any one composition (four or five minutes for the most complex), 
and most compositions need to be scored by only one rater. An important 
side effect of such training (where it has been studied in laboratory like 
settings in school districts) is that the teachers so trained give students 
more writing assignments and, subsequently, writing performance improves. 

A final concern with writing assessment involves the comparability of 
assessments. Does students' performance wobble annually depending upon the 
topic, the mode of discourse, e.g., exposition vs. narrative, or the 
scorers' orientation? Implemented in this study are procedures to permit 
the development of sets of task examples as well as methods to determine 
the extent to which performance within students differs depending upon the 
writing tasks. 

The study was designed, then, to address all of the concerns above. 
It was further refined and improved to reflect the issues and experiences 
of other scholars in different countries. In particular, a developmental 
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sequence of writing tasks was proposed and, to a reasonable extent, will be 
ultimately tested by performance drawn from the three grade levels 
described. The actual concepts, objectives, procedural tasks, and measures 
are described in later portions of this report. 
Background to the USA Study 

The International Study was conceived by Alan Purves, at the 
University of Illinois. Early in 1980 he and staff of the UCLA Center frr 
the Study of Evaluation collaborated on a proposal to be submitted to lEA 
for approval. Purves brought to the task understanding of the 
international community, of English language curriculum, and of the general 
products that might be usefully developed. Eva Baker, Edys Quellmalz, and 
Frank Capell, then all of CSE at UCLA, provided technical support in the 
design of domain-referenced writing specifications, alternative analytic 
rating research, descriptive studies of rater training and rater scoring 
stability, and psychometric analyses of writing. Merging these two 
perspectives was not automatic but made easier because of the joint 
agreement on the importance of the study, prior collaboration between UCLA 
and the University of Illinois, and the backgrounds of UCLA participants in 
the teaching of English. 

As the proposed activity became more specific, with the full 
participation of an International Steering Committee, the UCLA role became 
focussed on the USA Study. Originally, funds for this study were thought 
to be available through other governmental sources. Subsequently, a 
self-supporting model, in which 8*10 states would serve as replicates, was 
planned. Unfortunately, this plan did not truly satisfy the lEA need for 
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national representation, and furthermore, many states had slightly 
different views of which writing tasks were important. (An important 
side-effect of this effort is that conceptually comparable writing 
assessment activities, reflecting a subset of the tasks proposed here, have 
been undertaken in Maryland, Illinois, Connecticut, and California,) 

In the sections which follow, the history of the development effort is 
summarized and an overview of the Study of Written Composition is 
provided. 

To date, the principal support for the study came f y^i The Spencer 
and The MacArthur Foundations and the NIE, with additional resources from 
the University of Illinois, the Center for the Study of Evaluation, UCLA, 
NIE, and T. ;i National Council of Teachers of English, among others. 
Accomplishments to date include: 

1. An overall conceptual framework was developed. 

2. A study design was developed, approved and implemented. All tasks 
were developed and piloted. 

3. All questionnaires were developed and piloted. 

4. Scoring guides were developed. 

5. International scorer training was completed. 

6. Sampling was conducted with assistance from the National Center 
for Educational Statistics. 

7. Support was secured from the national government, educational 
policy makers, professional organizations, and leading individuals 
in measurement, evaluation, and English language teaching, and 
most significantly, from the Council of Chief State School 
Officers. 

8. School contacts, data administration, and data return were 
accomplished. 

9. Raters were trained. 

10. Scoring of tasks was accomplished. 

11. Preliminary data analys*is of outcomes was conducted. 
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Overview of the lEA Study of Achievement in Written Com r ;sition 

The teaching of written composition ib a good example of a research 
area where both theoretical and empirical work is very much needed. The 
domain of writing is not well defined and, therefore, test construction is 
also at an early stage of development. In addition, tiiere has been 
clear-cut agreement neither on the criteria of good writing nor on the 
methods of scoring. In this respect the Study of Written Composition 
differs clearly, for instance, from the Second Mathematics Study or the 
Second Science Study. During the first International Study Committee 
meeting it also becanie evident that the content and methods of instruction 
in written composition are to some extent unknown even in participating 
countries themselves. 
Aims of the Study 

The foreqoing discussion of the current situation suggests that an 
important aim of the study is to provide a conceptualization of the donfiain 
of .vriting in general and of school writing in particular. This is a 
necessary step for the construction of a ret of writing ta^rks which can be 
justified! both theoretically and in terms of curricular validity. 

The foregoing discussion also suggests that another important aim of 
the study is the exploration and ooscription of (a) what is being taught in 
the instruction of written composition, (b) how wiM'tten composition is 
being taught, (c) what kinds of exe*^cises and assignments students are 
given in tests and examinations, (d) how much time is devoted to written 
composition, and (e) what characteristics are valued and what criteria are 
used in assessing performance in written composition. 
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A thirc' major aiin of the study is to describe how students in 
different countries respond to the assignments . This will include an 
attempt to score compositions using an internationally agreed upon scoring 
scheme as well as a more qualitative description of general patterns and 
variations in repsonse that might he associated with ceitain schools of 
thought and/or certain cultural variations. 

A fourth aim of the study is to test some hypotheses about factors 
that are assumed to affect performance in written composition. These are 
based on a model developed for the study (see Figure 1). 
Research Design 
Populations and Samples 

The definitions of the populations are: 
Population A : Students at or near the end of primary education and the 
self-contained classroom. 

Po pulation B : Students at or near the end of comprehensive education, 
i.e., students who are in the last year of the shortest secondary program 
and those in longer programs who have completed the same number of years of 
schooling whether or not they have finished their program. 
Population C : Students at or near the end of academic secondary school. 
Each National Center will draw a representative stratified sample of 
students using strata that are important from the national point of view. 
International Instrument s 

Data for the study was collected by means of a series of instruments 
specially prepared for the study — writing ^sks, questionnaires, and some 
attitude measures. Teachers administered all instruments. All writin g 
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Figure 1. Gsneral Model of tJie Study (Main Constructs) 
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tasks were direct writing samples, in other words, no indirect "objective" 
tasks were used in the international component. Tasks were chosen on the 
basis of our model for the specification of the domain of writing and the 
information provided by participating countries on curricula, typical 
topics and writing tasks used in participating countries, and information 
about and samples of examinations. 

There were some common writing tasks across the three populations (see 
Table 1). Tasks were rotated in order to cover a wide area of appropriate 
writirig tasks without extending testing time too much. 

National background information was obtained by means of the 
Curriculum Questionnaire , the Interview Schedule , National Context 
QuestiOfmaire , and National Case Study Questionnaire . 

Attitude questions were used to measure students' attitudes towards 
school and writingc 

The T eacher Questionnair e was given to all the teachers who teach 
written composition to the students in the samples, to obtain information 
on teachers* qualifications, experience, teaching and feedback methods, 
etc. 

The School Questionnaire was answered by the school principals and 
will ultimately provide data on the community and the school. 

As a general principle, countries were al'^^ encouraged to add other 
types of national options to the basic international components, provided 
that the latter was not jeopardized, so that the total testing program In 
any given country (1) was perceived as appropriate by students, teachers 
and other interested parties, and (2) provided information that was 
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relevant in terms t>f national problems in the instruction of written 
composition. Because of cost constraints, no U.S. national options were 
exercised. 

Sampling of Participa V^ nj Schools 

Of critical concern in this study was assuring a representative sample 
of respondents. This problem was complicated because the request for 
student time required two dayb of partic^nation, far more than the usual 
"scientific survey." Furthermore, because participation was a local 
option, we were sensitive to the relatively low response rates achieved by 
studies of this sort. Third, because of the delay in funding for this 
effort, we would not have the leisure to engage in extensive follow-up or 
resampling. Therefore, our sampling strategy included leadership, 
practical, and technical concerns. 

Leadership incentives. We believed that it was imperative that the 
5chools be willing to participate. We did not have sufficient funds for 
lengthy, persuasive conversations. We secured the participation of the 
Council of Chief State School Officers whose leadership lent strong support 
to the idea of inte»^national comparisons and was also committed to 
improving the database of national indicators of educational quality. The 
timing of our request, following as it did national reports on USA 
educational performance, was fortuitous. We were able to transmit to each 
Chief Officer of State Education Departments a letter that described the 
study, the list of sampled districts in each particular state, the UCLA 
request letter to the district and schools for participation, and a draft 
letter to be sent with the Chief's sig.idture, urging schools in our sample 
to participate. We believe that this step of securing the support of state 
leadership was critical t) the success of the study. 
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Practical incentives ^ On the practical level, we also felt it 
essential to provide the local schools with motives to devote valuable 
instructional time to this study. In our letter, we stated our intentions 
to provide school level data back to the sending schools, both in terms of 
performance on each of the writing tasks and also with regard to national 
and international performance levels (when they were available) > We 
focussed our efforts on the improvement of writing ability and also 
intended to provide analyses related to what particular instructional 
implications might be drawn from their school's results. We also promised 
to provide models for developing similar writing tasks and procedures for 
scoring writing according to the standards used in the study. In addition, 
we believe that we vjere assisted by the perception of writing as un 
important educational goal. We believe participation wis forthcoming 
because school personnel could s<^e the educational value of asking students 
to vi^rite, the major focus of our study. 
Technical Concerns 

In order to achieve the three objectives of the lEA study, as stated 
above, it was necessary to draw a sample which would represent the entire 
count'T^y for each of the specific populations. A variety of alternative 
approaches were considered to accomplish this representativeness; however, 
previous experience with the lEA Study of Science and available information 
on Che total populations dictated the final approach. The remainder of 
this section discusses our approach, organized around three topics. The 
initial topic is the sampling of participant schools. This is followed by 
a description of the replacement strategy and procedure. And finally, a 
description of the procedure for selecting classes is presented. 
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The Selection of Schools 

The approv :hoseri for the study was to draw a stratified 
probability-proportional-to-size (PPS) sample for each of the three popula- 
tions. This approach produces a random sample that is representative of 
the population and allows for estimation of national proficiency levels. 
In order to implement the sampling, it was necessary to obtain information 
on the total U.S. populations of schools. This information was provided by 
the National Center for Educational Statistics*(NCES) which allowed 
access to their data base describing public and private schools within the 
United States. It should be noted that the available information on the 
public Id private schools differed both in its recency and its content. 
The public school information was derived from a survey current through the 
1982 school year while the private school information was current through 
1980. However, it was felt that the difference In recency and type of 
information could be minimized by the design of the sample and sampling 
procedure. 

In drawing a stratified PPS sampie for each population, the 
stratified':'' on dimension selected was the type of school (i.e., public or 
p-ivate). Other potential dimensions for stratification such as community 
type, socio-economic status, and student racial/ethnic composition were 
also considered but discarded. Experience with previous lEA studies had 
indicated that, if the sample size is sufficiently large, these dimensions 
will be adequately represented in a sample drawn through a simple random 

*We would like to express our appreciation to NCES, and particularly to 
Dr. Larry Suter, for their cooperation and assistance with this aspect o^ 
the study. 
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selection procedure. Since the smdn3st sample for the current study was 
over a 100 schools, it was felt that reliance on random sampling b> school 
type was sufficient. The choice of this approach was also consistent with 
the available information at NCES where the selection was to be performed. 

Procedurally, the sampling required the creation of a computer routine 
which would enter the appropriate data base, compute a measure of size for 
each school, order the schools with respect to this measure, and then 
select every "Nth" school as a sample participant. The measure of size 
used in the sampling was an estimate of the number of students enrolled in 
a school at the target grade level. The estimate was derived by dividing 
the reported school enrollment by the reported number of grade levels 
served by the school. This estimate assumes uniform distribution of 
students over grades which while potentially inaccurate in some cases was 
the most appropriate procedure for estimating size given the available 
data. 

In addition to ordering the schools on size, schools were also ordered 
by state. While this, the secondary ordering did not represent a full 
stratification it increaseo the geographic coverage of the sample. 
Geographic coverage was viewed as desirable since it would enhance the 
likelihood of national representativeness, spread the respondent burden 
across states, and reduce the burden at the district level. 

After schools were ordered on size and state, two additional pieces of 
information were required to perform the selection: a random start value 
and a constant interval size. The former was drawn from a table of random 
numbers. The latter was calculated for each stratum by dividing the sum of 
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the measures of size by the desired number of schools for the sample.* 
When these two pieces of Information were provided to the computer 
routine**, a list of sample schools was produced. This procedure was 
repeated for each stratum and for all three populations. 
Selection of Replacement Schools 

In addition to drawing the primary sample for the study, it was 
necessary to anticipate the need for replacements should any of the 
participant schools decline to participate In the study. It was decided 
that the most efficient method for handling the replacements was to draw 
parallel samples. This approach was used succesfully in the lEA Science 
Study and decreased the handling and logistical problems involved with 
replacements. Since the timeline for the conduct of the U.S. study was 
exceedingly short, it was felt that these logistical constraints outweighed 
any other considerations. Thus, a total of four samples for each stratum 
within each population was drawn. This Included a primary sample and three 
replacement waves. 

Actual use of the replacement schools was made on a case by case 
basis. That is, the procedure for replacement was initiated only after a 
school refused participation and there were insufficient participating 
schools to satisfy the state quota derived from the primary sample. The 
state quota was set by the number of schools in the primary sample that 
originated from each state. When the need for replacement arose, schools 

*The sum of the size measure was computed from previous analyses of the 
total school populations. 

**The computer routine used for the sampling wa? a modified version of 
that used at NCES for the lEA science study. 
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from the same state and from the first available replacement wave were used 
as potential replacements. Final selection was performed by selecting the 
schools on the basis of the measure of size so that the replacement was as 
similar as possible to the non-participant. 

It should be noted that prior to the selection of the four sample 
waves for each population certain exclusions were made frorr the total oool 
of schools contained in the NCES data base. Specifically, schools were 
excluded if they were insti tions which served special non-normal 
populations of students, Ci if they were not operating in the most recently 
reported year, or if they were located in other than the 50 states and the 
District of Columbia. It was felt that these exclusions were consistent 
wfth the intent of the study and insured high data quality. 
Selection of Classrooms 

The last aspect of the general sampling approach was the specification 
of the particular classroom to participate in the study. The design of the 
international study called for one classroom to participate from each 
selected school at the target grade level. To facilitate identification of 
the classroom at the school level, it was decided that a simple but appro-^ 
priate method of selection must be employed. Therefore, it was decided 
that the class should be selected on the basis of size. Since only one 
class was needed at each school, the school had only to identify its 
largest class from the appropriate grade as the participating class. This 
approach was consistent with the strategy chosen for the selection of 
schools and greatly minimized the logistical problems of class selection. 
Implementation of this process was left to the School Coordinators, 
although they were assisted by a description of the procedure in the 
Coordinator Manual provided to them. 

20 
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To summarize, the general sampling approach addressed the issue of 
national representation while balancing const»^aints of available informa- 
tion and logistics. The resultant approach ms to draw a stratified PPS 
sample from two strata In each of the three populations. The strata were 
defined a*^ public schools and private schools. Within each stratum, 
schools were ordered by size and state prior to selection. Four parallel 
samples were drawn for each population and from within each stratum. This 
strategy allowed for easy replacement while insuring the same sample struc- 
ture as in the primary sample. Classroom selection was also performed 
using size as the selection criteria. Since only a single class per school 
was needed, this approach resulted in the selection of the largest class at 
the target grade. 

The remainder of this document addresses the aspects of the sampling 
plan and procedure which were unique to the three individual populations. 
Specifically, each of the following three sections provides a definition of 
the population and a £\)ecification of sample size. 
Population A: Grade Six Schools 
Definition of Target Population 

PopuUition A is defined as students at or near the end of primary 
education and in a self-contained classroon. For most states, this occurs 
during the sixth grade and the median age of tne students is 11 years. 
Because, the end of compulsory schooling in each of the participating 
states is at age 16 years, virtually lOOJ of sixth grade students in the 
population attend school. 

(1) Desired target population: all sixth r^'ade students. 

(2) Defined target population: all sixth grade students on the 
dates of testing. 

21 
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(3) Excluded population: all students above or below Grade 6 and 
all students not in school on the dates of testing. 

Sample Size 

The international study design requires a sample of at least fifty 
classrooms for this grade level. The United States study was designed with 
slightly more than double this sample size thus providing greater precision 
for national estimates. Specifically, the Population A sample was designed 
to include 100 schools from the Public School stratum and 10 schools from 
the Private School stratum for a total of 110 grade six schools. The 
allocation of the total sample to the two strata was a funcc^'cn of interest 
and resources. It should be noted that a 25% oversampling of the Public 
School stratum and a 501 oversampling of the Private School stratum was 
actually performed from this population to help insure that at least the 
required number of schools would be available for participation. That is, 
since the information in the data base was dated it was anticipated that 
some percentage of th^^ sampled schools would be ineligible for study 
participation (either because the school no longer existed, or because the 
school no longer served the grade level of interest). This procedure 
resulted in 125 public schools and 20 private schools in each sample wave. 
Our target was 110 schools. 
Population B: Tenth Grade Schools 
Definition of Target Population 

Population B is defined by students who are at the end of compulsory 
schooling. The end of compulsory schooling in each of th^ participating 
states is at age 16 years. In general, students become 16 years old just 
after or during the last half of their tenth grade year. Thus, most tenth 
grade students are in the last year of compulsory education. 
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(1) Desired target population: all tenth grade students. 

(2) Defined target population: all tenth grade students on the 
dates of testing. 

(3) Excluded population: all students above or below Grade 10 and 
all students not in school on the dates of testing. 

Sample Size 

The international study design requires a sample of at least 100 
classes at the tenth grade level. The United States study was designed for 
slightly more than double this number to a total of 220, The rationale for 
the larger sample was the greater precision to be achieved in the popula- 
tion estimates to be derived from the sample. The Population B sample was 
structured so that 200 public schools and 20 private schools would partic- 
ipate in the final study. As In Population A, a 25% oversampling of the 
Public School stratum and > SOX oversampling of the Private School stratum 
was performed. This resulted in 250 Public Schools and 40 Private Schools 
In each of the four Su^nple waves. 
Population C: Twelfth Grade Acadeaic Students 
Definition of Target Population 

Population C is defined by students who are at the end of an academic 
program in secondary school. For the states participating in this study, 
this occurs in grade 12 and the mean age of 12th grade students is 17 
years. 

(1) Desired target population: all twelfth grade students in 
academic programs. 

(2) Defined target population: all twelfth grade students in 
academic programs on the dates of vesting. 

(3) Excluded population: all students above or below Grade 12, 
all students not in school on the dates of testing, and all 
Grade 12 students not on academic tracks. 
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Sample Size 

The international requirements for this population are the same as 
that of Population A, namely 50 classes. The United SUtes study differs 
from that by slightly more than double. Specifically, a total of IJO 
schools was selected as the desirable number of participants for the U.S. 
study. The allocation of these across strata was the same as for Popula- 
tion A with 100 public schools and 10 private schools. It should be noted 
at this point that ^he schools selected for participation in the Population 
C study were limited to those schools which were participating in the 10th 
grade study. Thus, the final selection of schools for Population C was 
linked to the Population B sampling with that population (B) providing the 
pool of eligible schools for the Population C samp'Je. As with the previous 
populations a 25% oversampling of the Public School stratum and a 50X over- 
sampling of the Private School stratum was performed. This produced 125 
public schools and 20 private schools in the four sample waves. 
Response to Sample 

The results of the sampling plan were encouraging. The lEA minimums 
for 6th grade were 50 classrooms, for 10th grade, 100 classrooms, and for 
12th grade, 50 classrooms. Our intention was to double the minimum sample 
sizes, so our functional target? were 100 cla^^rooms at 6th grade, 200 at 
10th grade, and 100 at 12th grade, although our sampling procedures used 
110, 220, and 110 as desired samples. 

As noted earlier, we oversampled by thr reported percentages to expe- 
dite the data collection process. In Table 1 below, we report the actual 
number of classrooms included in our scoring and preliminary analyses. 
Note that private schools are slightly over represented. These figures do 
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not include data from three classrooms which arrived too late to be 
scored* 

Table 1 

Classroom Data at Each Grade Level 



SCHOOLS 


6TH GRADE 


lOTH GRADE 


12TH GRADE 


TOTAL 


PUBLIC 


72 


167 


92 


331 


PRIVATE 


8 


26 


11 


45 


Totals 


80 


193 


103 


376 


Percent of 
Targets 


802 


96+i 


1032 


942 



Twelfth grade classes were selected from schools with 10th grade 
participating classrooms. This decision was to facilitate (reduce) the 
number of school contacts required, and more importantly, to provide a 
school level factor for data analyses* Table 2 below, therefore presents 
the number of unique schools included in the study analyses* 



Table 2 

Total Uniaue Schools Providing Data 



PUBLIC 


PRIVATE 


TOTAL 


POPULATION 


A: 


{6th grade only) 


72 


8 


80 


POPULATION 


B: 


(10th grade only) 


75 


15 


90 


POPULATION 


C: 


(10th and 12th grades) 


92 


11 


103 


TOTAL 






239 


34 


273 
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Siorirg and Ttaining 

Logistics for training raters and scoring data were enormously 
complex. Resources were not available to score all papers. After 
conferiMng with NCES and NIE personnel, it was egreed to score eight papers 
per task ^or each classroom. Actual papers scored vary because of student 
absences and differential class enrollments, and because of the complex 
rotation patterns for tasks, within classroom sampling by task occurred. 
Scoring Approach 

lEA scoring procedures were followed. Essentially these required 
scoring on three analytic subscales, organization, style, and content 
detail and appropr. *eness> and a general impression scoring. Task 
differences also influenced patterns of training and scoring. Ke scored 
similar tasks (lA-E, 6^ 7) in the first scoring session held in June, 1985. 
and tasks 3» 5, 9 in ^he second session. In addition, 20% of the papers 
were scored by two separate raters (double scored). We also used check 
papers (20 per task per day) to assure that raters were staying on the 
scale. In addition, our plan required the scoring of calibration papers 
(to permit equating for international comparison). Calibration papers were 
scored by everyone for each task. Approximately 20 calibration papers were 
used for each of the 10 tasks. 
Selection of Scoring Directors and Raters 

A list of potential scoring directors and raters was devised based on 
our collective experience in the fielf^ of writing assessment. We also 
sooght recommendations from districts throughout California as well as from 
other university institutions both within UCLA and at other universities. 
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Scoring directors were selected on the basis of their experience in 
using an analytic scoring scheme or in participciting ds a trainer in large 
scale writing assessment efforts. Raters were also selected on the basis 
of their experience as rater*^ in large scale or district wide scoring 
efforts. However, it was not mandatory that they be familiar with the 
analytic types of scoring rubrics. Public and private school personnel 
formed the rating pool. For the June session, 40 raters were used; for the 
August session 20 raters were used. 
Valication Scoring 

A validation meeting for the J ^3 scoring session was held including 
two CSE coordinators and four scoring directors. This meeting had several 
purposes: 1) to acquaint the scoring directors with the lEA stu(|y and with 
the procedures for training scorers and scoring tl^e writing samples; 2) to 
review the scoring rubric for each writing task to be scored and make any 
necessary clarifications; 3) to select and score training, qualifying and 
check papers. 

In preparation for this meeting, the CSE coordinators selected a pool 
of papers for each task which represented various proficiency levels and 
scoring problems. These papers were selected from the larger sample to be 
scored* The validation meeting resulted in a pool of scored training, 
qualifying, and checkpapers, complete with scores and comments. These were 
later xeroxed and prepared for the purpose of training the raters. 

Prior to the validation session notebooks were devised for the scorino 
directcrs. These contained a list of potential training papers, qualifying 
papers, and checkpapers as well as copies of each type of paper. A copy of 
the scoring manual and scoring sheets were also included. Each scoring 
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director was provided vrith a notebook containing the information 
corresponding to the task(s) which she was responsible for. After the 
validation similar notebooks were devised for the raters. These included a 
list and copies of the training papers and a list and copies of checkpapers 
for the first day. 

Validation of training, qualifying and checkpapers for the August 
session was conducted via the mail and telephone. This was possible 
because the scoring directors as well as the raters wer^ well acquainted 
with all aspects of the training and scoring session because of their 
experience in the June scoring session. 
Rater Training and Scoring Sessions 

Orientation. The orientation was conducted by the CSE Coordinators 
during the first morning of the session. Raters were introduced to the 
coordinators, scoring directors, and support staff. They received an 
overview of the lEA study and of the general training and scoring 
|. ^wcdures. 

Afterward the raters were assianed to their training and scoring 
group. In making the assignment, coordinators spread groups of raters who 
came from the same district across scoring groups. The intent here ws to 
ensure that personnel coming from the same district would take with them 
the variety of scoring skills offered by the lEA training and scoring 
sessions. 

After the overview and assignment of groups, the scoring directors 
began to train their raters on the interpretation and applicatic; of the 
lEA scoring rubric to the specific writing tasks and samples to be scored. 
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Rater training . After giving everyone the opportunity to carefully 
read the scoring criteria for a particular writing task, the trainers 
(using overhead transparencies) illustrated each score for each scoring 
element with a writing sample. After this had been done for all of the 
scoring criteria and range of scores for a particular task (for example, 
task 6), raters were given t^»^ opportunity informally to score a few (3-5) 
training papers. The trainers took a voice count and tallied the scores 
for the training papers to get a sense of how well everyone was applying 
the rubric. These scores also signaled potential misunderstandings or 
unusual aspects of the paper s< The trainers and raters discussed these, 
giving examples to clarify the problems. The process of scoring training 
papers and discussing the scores continued until the trainers felt that the 
raters were ready to pass the qualifying papers. Agreement was defined as 
within one score point on the six point scale for each scale scored. The 
qualifying papers were regarded as a test to determine the extent to which 
individual raters were accurate, as well as the extent to which the entire 
group of raters (for a particular task) agreed with each other. If a 
rater, or the group as a whole, did not pass the qualifying set, they were 
retrained. To pass the test, both individually and as a group, the scores 
reflected an agreement of at least .80. For example, if there were 10 
qualifying papers for task 6, then each rater was allowed only two errors 
for the entire set of papers across all of the scoring elements. In both 
the June and August sf'ssions, all groups passed their qualifying sets on 
the first administration and no group retraining was necessary. 
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After passing the qualifying sets, raters went on to score the sample 
papers. However, to ensure that high reliability was maintained both 
individually and for the group, a set of approximately 20 checkpapers had 
to be scored on a daily basis; half of these were scored in the morning and 
half in the afternoon. Again^ 802 agreement was the standard. As 
necessaryp the individual raters were taken aside to discuss the particular 
errors they had made, and then retested. 
Preliminary Data Analysis 

Data analysis is presented for Tasks la, lb, Ic, Id, le, 6, and 7. 

Initially, a total of 11,131 essays were scored for the above seven 

tasks. Ir. this first cleaning run, 2.20 percent (245 cases) of these 

cases were lost due to missing or invalid identification numbers, reducing 

the number of cases to 10,886 for the seven tasks. In the final cleaning 

run, 6.23 percent of the data was lost to missing grade levels, population 

level (public vs. private), and missing task numbers (a total of 678 

cases). The final data set contains 10,208 essays (see Table 3). 

Table 3 
Deletions from Data Set 



Initial Essays Scored: 
Deleted Due to ID's 
Other Deleted Data 

TOTAL DATA SET 



ESSAYS PERCENT 

11,131 100.00 

245 2.20 

678 6.09 

10,208 91.71 



Table 4 presents distributions of essays in final data set by task, 
grade, and public or private school population. Table 5 displays the total 
essays scored by task across grade levels. 
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Table 4 

Essays In Final Data Set 
by Task, Grade and Population 

6th Grade 





lA 


IB 


IC 


ID 


IE 


6 


7 


PUBLIC 


355 


301 


293 


370 




559 




PRIVATE 


26 


27 


27 


27 




39 




TOTAL 


381 


328 


320 


397 




598 





10th Grade 





lA 


IB 


IC 


ID 


IE 


6 


7 


PUBLIC 


882 


946 


870 




779 


737 


687 


PRIVATE 


149 


152 


156 




141 


143 


131 


TOTAL 


1031 


1098 


1026 




920 


880 


818 



12th Grade 





lA 


IB 


IC 


ID 


IE 


6 


7 


PUBLIC 










576 


815 


747 


PRIVATE 










79 


99 


95 


TOTAL 










655 


914 


842 



28 



Tabic 5 

Totals of Scored Essays by Task 



TASK 


lA 


IB 


IC 


ID 


IE 


6 


7 


TOTAL 


6th 
GRADE 


381 


328 


320 


397 




598 




?024 


10th 
GRADE 


1031 


1098 


1026 




920 


880 


818 


5773 


12th 
GRADE 










655 


914 


842 


2411 


TOTAL 


1412 


1426 


1346 


397 


1577 


2391 


1660 


10,208 



Preliminary Descriptive Statistics 

Descriptive statistics at the student level were computed by task and 
grade level. These analyses were computed for each of five outcome vari- 
ables: 01 - Overall Impression; COMP - a competency measure averaging 
three analytic scores; CONTENT - Content appropriateness and specificity; 
ORGAN - Organization; and STYLE. The scores ranged from 1-6. These are 
presented in the Tables 6-18 below. 

Table 6 

Means and Standard Deviations for Task 1 by Task and Grade 
Task » lA Grade = 6 



VARIABLE N MEAN STD DEV 



01 380 2.765 0.907 

COMP 380 2.807 0.831 

CONTENT 380 2.702 C.943 

ORGAN 38C 2.936 0.908 

STYLE 380 2.781 0.931 
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Table 7 



Means and Standard Deviations for Task 1 by Task and Grade 
Task « lA Grade = 10 



VARIABLE 


N 


MEAN 


STD DEV 


01 


1035 


3.2516 


0.952 


COMP 


1035 


3.275 


0.885 


CONTENT 


1035 


3.283 


1.071 


ORGAN 


1035 


3.339 


0.903 


STYLE 


1035 


3.202 


1.033 



Table 8 

Means and Standard Deviations for Task 1 by Task and Grade 
Task = IB Grade = 6 



VARIABLE N MEAN STD DEV 



01 328 2.896 1.194 

COMP 328 2.970 1.122 

CONTENT 328 2.817 1.257 

ORGAN 328 3.146 1.129 

STYLE 328 2.948 1.181 



Table 9 

Means and Standard Deviations for Task 1 by Task and Grade 
Task «= IB Grade «= 10 



VARIABLE 


N 


MEAN 


STD DEV 


01 


1104 


3.411 


0.917 


COMP 


1104 


3.440 


0.854 


CONTENT 


1104 


3.411 


1.034 


ORGAN 


1104 


3.501 


0.872 


STYLE 


11J4 


3.409 


0.924 
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Table 10 

Means and Standard Deviations for Task 1 by Task and Grade 
Task « IC Grade « 6 



VARIABLE N MEAN STD DEV 



01 320 3.003 0.990 

COMP 320 3.004 0.953 

CONTENT 320 2.956 1.058 

ORGAN 320 3.121 0,976 

STYLE 320 2.934 0.037 



Table 11 

Means and Standard Deviations for Task 1 by Task and Grade 
Task « IC Grade = 10 



VARIABLE N MEAN STD DEV 



01 1031 3.491 0.876 

COMP 1030 3.497 0.851 

CONTENT 1031 3.465 0.999 

ORGAN 1031 3.568 0.879 

STYLE 1030 3.457 0.933 



Table 12 

Means and Standard Deviations for Task 1 by Task and Grade 
Task « IE Grade « 10 



VARIABLE N MEAN STD DEV 



01 927 2.881 0.917 

COMP 927 2.800 0.827 

CONTENT 927 2.555 0.949 

ORGAN 927 2.907 0.920 

STYLE 927 2.938 0.887 
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Table 13 



Means and Standard Deviations for Task 1 by Tark and Grade 
Task = IE Grade « 12 



VARIABLE 


N 


MEAN 


STD DEV 


01 


656 


3.135 


0.876 


CCMP 


655 


2.986 


0.779 


CONTENT 


656 


2.711 


0.938 


ORGAN 


655 


3.068 


0.898 


STYLE 


655 


3.180 


0.800 



Table 14 

Means and Standard Deviations for Task 6 oy Grade 

Grade « 6 



VARIABLE 


N 


MEAN 


STD DEV 


01 


705 


2.561 


1.624 


COMP 


664 


2.382 


1.168 


CONTENT 


664 


2.296 


1.232 


ORGAN 


664 


2.246 


1.209 


STYLE 


664 


2.603 


1.218 



Table 15 

Means and Standard Deviations for Task 6 by Grade 

Grade = 10 



VARIABLE 


N 


MEAN 


STD DEV 


01 


1150 


3.099 


0.959 


COMP 


1143 


3.08C 


0.815 


CONTENT 


1K4 


3.124 


0.968 


ORGAN 


1143 


2.928 


0.922 


STYLE 


1144 


3.202 


0.805 



o 
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Table 16 



Means and Standard Deviations for Task 6 by Grade 

Grade > 12 



VARIABLE 


N 


MEAN 


STD DEV 


01 


1046 


3.530 


0.986 


COMP 


1039 


3.474 


0.848 


CONTENT 


1039 


3.527 


0.987 


ORGAN 


1039 


3.390 


0.963 


STYLE 


1039 


3.505 


0.836 



Table 17 

Means and Sta d Deviations for Task 7 by Grade 

Grade = 10 



VARIABLE 


N 


MEAN 


STD DEV 


01 


1045 


3.039 


1.011 


COMP 


1036 


3.032 


0.832 


CONTENT 


103t5 


3.031 


0.981 


ORGAN 


1036 


2.938 


0.954 


STYLE 


1036 


3.127 


0.799 



Table 18 

Means and Standard Deviations for Task 7 by Grade 

Grade = 12 



VARIABLE N MEAN STD DEV 



01 1014 3.339 0.985 

COMP 1005 3.308 0.815 

CONTENT 1005 3.325 0.975 

ORGAN 1005 3.241 0.919 

STYLE 1005 3.357 0.792 
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Analyses of variance were computed for those tasks assessed at more 
than one grade level. These analyses are presented In Table 19 below. 
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Table 19 

TASK«tA 

ANALYSIS OIF VAHIANCC PP0CE0UR6 
CLASS LEVCi. INFCRNATIUN 
CLAbS LCVeCS VALUES 

CRAOC 3 ABC 



MUMOLM OF UOSERVATtUNS In BY CROUP ■ 1420 



5A3 
TA!>K*1 A 

ANALYSIS OF VARIANCE PROCEDURE 





OLPbNJbNT 










SOUHCE 




OF 


SUM UF SQUARES 




MUDCL 




Z 


78»J4442b70 




~ ERRJR 




i 413 


I2S0»0U34274I 




CUKWLCTbO 


tutal 


1 41b 


i J2tt«4J7tfb Jl 1 




SUUHC£ 




OF 


ANUVA JS 




GMAJC 




2 


7d« J4442570 



MEAN SQUARE 
3V« 17221209 
0« 66470872 



F VALUE 
44»2a 



Pft > F 
O^tOOOl 
ROOT MSe 
0»940S8«'>b 



P-SQUARE 
0«0*itfV75 



29* 622 
01 MEA.M 
3*1539546 



F VALUE 
44t2i) 



PR > F 
0*000 I 



BEST COPY 

3S 



TASKstd 

ANALYSIS OF VARIANCE PRUCEUURE 
Ci./'SS l.UVk;L INFORMATIPN 
CLASS LCVCLS VALUES 

GHAOE 2 AO 

NUflDCR OF UOSCRVATIONS 'N BY CROUP « 1436 

NUTti: Ai.L OtZl>cNO NT VAHi AdLES AMt£ CJNSl:»TENr tflTH RE;>PCCi TO ThE PHESENCC OR AOSENCC OF MlSSlNO VALUliS* HU»CVCR« ONLY 14J2 
UdSCRVATlU^:* IN bY bftUUP CAN UL USUI) IN THIS NALYSIS* 



UEP^NULHT VAt^iAULL: Ul 

SJuncc t>F 

MODEL i 

ERROR M30 

CORRECTCe TOTAL l^JI 



SOURCE 
GRAOE 



DF 
1 



SUM OF SUUAKCS 
67«039J0«U6 
1395*77033439 
|462«dlS04246 

ANUVA 
67*03930006 



SA'» 
TASKiiie 

ANALYSIS OF VARIm 



MHAH SUUA^tL 
O7«03 .30406 
0*97606737 



FRUCCDURC 



F VALUE 
6a*68 



PR > F 
0*0001 



F V ue 

66*06 



PR > F 
0*0001 
ROOT HSE 
0*Vd796122 



R^SQUARE 
0*045629 



C*V* 
29*999. 

01 mtfkii 

3*2932960^ 



BEST copy ^ ^ 



5AS 
TASK«1C 

ANALYSIS or VArtlANCC f»|;iOC£OUP£ 
CLASS LtSVEL INPCRMATIQN 
CLASS LEVCLS VALUES 

CkAUC 2 A e 

NUMUCR OH OU:it:RVATlQNS Ih UY GHOUH • lJb4 



note: all DL^'wNULnT VAHIAULCS arc CUNUISTLNT itflTH RC:»PCCT TU THI: *»KCSCNCC 0^ AOSCNCE OF MISSING VALUCS* HOUdVhRt ONLY 1351 
UdSLKVAT 10N«» In UY \»RJUH CAN UL U'StO 114 THIS ANALYSIS* 



OEPb.%;;ihNT VAHlAbLt; 01 

SOURCU Of 

HOOCL I 

ERROR 1 349 

COii^rtLCrt.0 TJTAL 13^0 



SOUMCC 



OF 
I 



SUM OF SQUARCS 
98«30617ai 7 
It OA •67679741 

lt62*9d2V7S!>7 

ANOVA SS 
Sa«J0ol701 7 



SAS 
TASK>1C 

ANALYSIS OF VARIANCE PROCEOUfE 



MCAK SQUARE 
Sa«30617OI7 
0*01666569 



F VALUE 
71«20 



PR > F 
0«0C01 
ROOT MSE 
0«90492J03 



R«SOUAR£ 
0«0<50139 



26*604^ 

01 ME AM 
3*3760177 « 



F VALUE 
Tl«20 



PR > r 
0*0001 



42 



BEST copy 



SAS 
TASK«1£ 

ANALYSIS UK VARtANCC Pt^OCEUURC 
CLASS LKVCL INFCHMATtON 
CLASS LCVELS v;i.UES 

GRAOC J ADC 

NUHliEX OF Ob&ERVATlQNS IN UY GROUP m 1674 

NOTL: Ai.4. Ouf^NJLNT VAN! AULCS AKC C .NSISTLNT WITH RCSPL'CT TO TMCi PHESLNCC OH AOSENCE MlSStNC VAtUfcS. MOwEVkH. ONLY tSBtf 
iX>S4.HVATIUN:» iN UY ^ROUP CAU 5L Ui^tU iN IHIS ANALYSIS. »#i*.wc^. nww.iT«.r,. uni-T i^oci 



L 



UUPENaENT vahiaocl: ui 

SJUHCt Of 

HOOCL 2 

tiRRUR 1 SaS 

CURNECTbO TCrAL i 5d7 



SOUkCc 
GRAUt£ 



OF 
Z 



SUM UF SUUARLS 

134!»«<JSd3l2J4 
ANUVA SS 



SAS 
TASK«IE 

ANALYSIS OF VARlANCC PROCEOURC 



MEAN StiUARC 
16.0929J307 

o«a)8oa9tt7 



F VALUe 
19«67 



PR > F 
O.OUOl 
ROOT HSC 
0.90448321 



H^SOUARC 
0.024221 



C«V. 
30. 244 / 
01 MEA4 
2«9V0554i » 



F VALUL 
t9«o7 



PR > F 
0.0001 
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TASK* t A 

ANALYSIS Of VARIANCE PhxKtOUnt 
CLASS LCVtSL tN^UMMATIUN 
CLASS LCVtLS VALUES 

CRAOE 3 A » C 

KUMdLR Of UtiSCRVATlQNS IN i3Y UROUP • U20 



NOTbl ALL Ul:Huh«UUNT VAUi AdLC^ ARe CUNSISTLNT «ITH RLSf't-Ct Tu THC Pl^eSCNCE OH ABSENCE QF MiSSlNi. VALUCS* hUttft:VeH» QnLY 1416 
UUS£HVAr 1UN:> IH or group can Ut; KilitO IN TMlS ANALYSIS* 



UCPINJCNT VAHlAuLt: OjM** 

SOURCE DP 

MOOtiL 2 

CRROH 1413 

CuRRCCTLO total 1415 



sau»icc 

GlIAUL 



DP 
2 



SUM OP SQUARCS 
6k» 116:50706 
1073*33093765 
1134«447ti04n 

ANUVA :»S 
6I«116:>6706 



SAS 

ta:»k«ia 

ANALYSIS OF VARIANCE PROCEDURE 



MEAN SQUARE 
30»S5«!28393 
0«7996tl42 



F VALUE 
40,23 



PR > F 
0*0901 



F VALUE 
40*23 



PR > F 
0*0001 
ROOT MSE 
0*bndSod9 



R* SQUARE 
0*0:3J673 



t . 



C*V« 
27.677^ 
CO MP MEAN 
J*t490ll3. 
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BEST COPY 



00 
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SAS 

task«iu 

ANALYSIS OF VARIANCE PKOCi-OURE 
CLASS LEVEL tNFQRNATtUN 
CLASS LbVULS VALUES 

CHAOE 4 AO 



NUMbER OF OUSERVATiCJNS IN bY CiRuUP - U3o 

''"'•^^ -s.;^i::^^:i;i;^r::i•^J;S5p*^A^,%^*^l^l;''Mv """'^'^^^^ °« ^^^^^^ce ..ssinc valu^. Hu.evtR. only 



U3: 



UEP£Nut.NT VAi4lAdLc: uJMP 

SUURCL OF 

MUOEL t 

ERROR 1A30 

CURRLLTbO TuTAL 1 4J| 



S0U>4CU 
(*RAaE 



OF 
X 



SUM OF SOUARLS 

1210.10^09713 
12 74.1 llU4Jt>4 

ANOVA SS 
9ti.92<i936«)9 



SAS 
TASk«1 b 

ANALYSIS OF VARIANCE PROCEDURE 



MEAN SCuAHC 
5b«S2a9363« 
0.65187559 



F VALUE 



PR > F 
0.000 1 



F VALUE 
0!i.6S 



PH > F 
0.0001 
ROOT MSE 
0.92297107 



H* SQUARE 
0.043696 



c.v. 

27*691 1 
COMP MEA « 
3*3331 OOS 
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VD 



SAS 
TASKalC 

ANALYSIS Of VARIANCE PlOCeOUHB 
CLASS LCVCL iNFCHMAtlQN 
CLASS LCVLLS VALUES 

ORAOe 2 A B 

NUHliLft UF Ob&EHVATlQNS IN bY CkOOP « 1354 



NUTli: AcL JLPwNJLNT VAKI AiJLLS AHC CUNSISTENT WITH R'iSPLCt TO TH& PRk;SCNCe OR AUSENCC OF HISSING VALULS* H0«CV6Rf ONLY 
Utf^LMVATiUN^n IN dV liHUUP CAN UL UStU IN Til IS ANALYSIS* 



13S0 



L.. 

L 



SAS 
TASk«IC 

ANALYSIS OF VARIANCE PROCEDURE 



Ut:P£Nl>LNT ^Af^iAdCut 


CuMP 














SUUHCt^ 


HF 


SUM OF SUUARCS 


MEAol SQUARE 


F VALUE 


PR > F 


R-SQUARE 


C.V. 


MUOLL 


t 


99«47t>7c»735 


59*47676735 


77.31 


O.OOU 1 


0.054241 


439.V44i 


ERROR 


1 346 


1037.04471413 


0* 769321 00 




ROOT MSC 




COMP MEAI 


CORHECTbO TUTAL 


i 349 


l096»S2l4dl46 






0.07710946 




3«ja07407-« 


SUU>4CL 


UF 


ANUVA SS 


F VALUli PR > F 










0>4AaE 


1 


59«47u/67JS 


77.31 0.000 1 











BEST COPY 



lASK«t£ 

ANAUYSIS or VARIANCE PNUCCOUrC 
CLASS LCVt:L INFORMATION 
CLASS LEVCLS VALUES 

CRAOC 3 ABC 

NU»4UEH OF UQSkKVATlUNS IN bY UROVpP « 1674 

NJTC: ALL i)\J>uHut,HT VahI AdLES AHt CUNStSTi.NT »1TH RLSPECT TU THk: PRESCNCE OP AUSENCE UF NISSINQ VALUC^* HU«EVC*<« UNLY 
UUSCKVATIUNb IN UY uPUUP CAN JC XJiiLO IN THIS ANALYSIS* 



t5tf7 



L 



Dt^PENJLNT VAHlAbLi.: CJM^ 

SOURCi: OF 

MOO£L 2 

"ERROR t 584 

CaRRLCTl£0 TOTAL 1 9d6 



j SUURCt 
; GHAJi. 



OF 
2 



SAS 
TASK«t£ 

ANALYSIS or VARIANCE PROCEOURE 



;iUM OF SUUAkCS 

I04J«74293t73 
1067*63201008 

ANUVA SS 
23«Be407tt36 



MEAN SQUARE 
Ut«445J9te 
0*65892862 



^ VALUE 
I6«I3 



PR > P 
0 .000 t 
ROOT MSE 
0«8n74418 



R^SQUARE 
0*022376 



C«V. 
2«« Ib4«« 

COMP MEAU 
2«8b2l676l 



F VALUE 
t8cl J 



PR > F 
0*000 1 



|EMC 



BEST copy 



53 



ANALYSIS OF VAHIAKCC PJ^OCEOUHJ 
CLASS LbV£L INFURMATIQN 
CLAbS LevbLS VALUU!* 

UR*.-Z 3 A 8 C 

NUHbtH OF UbSERVAt|o»iS |N ^^y ^houP « U20 



L. 



SAi 
TASK* I A 

ANALYSIS Ur VARIANCE PHOrHOUPg 



OgPS£Ni>£NT 


VAhlAULn: 


UK<»aK 






3OUKC4: 






Sw , UF UaUAMdS 


MCAN StiUAME 


MOOeL 






4««496i^9V77 


23.2484494B 






A 413 


U96«44<f42792 




CORRCCTca 


TU1AL 


i 419 


l2O2.y4&J^7o0 




source: 




or 


ANUVA SS 


F VALUC l^fl > ^ 


GRAOli 




2 


4A.49btt9vr7 


2tf*4l 0*000 



^ VALUe 
2tf«4! 



PR > F 
OaOOOl 
ROOT MSE 
0.90467428 



R*SOUAHt 
0.038053 



28. 008 » 
ORGAN MCAN 
3.2302299-1 



54 



erJc 



BEST copy 



5 



SAS 
TASK* to 

ANALYSIS OF VARIANCE PROCEOURC 
CLASS cevCL INFCf^MATlQN 
CLASS LEVELS VALUES 

GHAOE 2 Ad 

NUMttLR or UUSERVATtUNS Ih OY GHQUP « U36 



SAS 
TASK«IQ 

ANALYSIS OF VARIANCE PRQCE0UR£ 



OtiPCNOCNT 


vAHi aule: 


UR^AN 














SQURCL 




OF 


OF SQUARES 


HEAh SQUARE 


F VALUE 


PR > F 


R*SUUARC 


C*V- 


MOUEi. 




1 


Ja • 992dV444 


31 •95299444 


36*35 


0<000l 


0*024790 


27*4tO 


ERROR * 




1 430 


l256»9719a«S7 


' 0*87900139 




ROOT MSE 




ORGAN MCAIi 


CURkkiCTLO 


TOTAL 


1431 


l2att»924;9dioi 






0*937b30f4 




3*4203910^ 


SOURCE 




Of 


ANUVA SS 


F VALUE PR > F 










(iHADE 




X 


3U9S2:i9444 


36.JS 0*000 1 











56 



BEST COPY 



57 



SAS 
TASKJilC 

ANALYSIS OF VARIANCE PKOCEUURE 
CLASS LCVEL INFCRNATION 
CLASS LCV£LS VALUES 

G^AOE 2 AO 

NUMBER C* OOSCRVATIUNS IH tfY GROUP • 1354 



NUTE: all JO'uNiiwNT VAKIAULCS ARE CONSISTENT «ITM RESPECT TO THE HRCSENCC OH AJSENCC OF MISSING VALUESt HOWEVER* ONLY U5I 
OdbcHVAT iON:> IN liY (tHOUP CAN USLJ IN THI:» ANALYSIS* 



OEPtiNOeNT VAMiAUCiiS CHbMN 
SOURCE OF 
MODEL t 
ERROR 1349 
COMrttCTtO TOTAL i 



SOUr^CC 
(*RAaE 



OF 
I 



SUM JF SHUAHES 
46»6862S429 
IIOI •176069ro 
U4y»662J24^0 

ANUVA :jS 
4tf •odb^^42S 



&AS 
TASK* I C 

ANALYSIS OF VARIANCE PROCEDURE 



MEAN SQUARE 
46t 66^25423 
0*61629064 



F VALUt£ 
Sy*64 



PN > F 
0*0001 



F VALUE 
S9*(>4 



PR > F 
0*OQ01 
ROOT MSE 
0*V034be04 



R«>SQUARE 
0*042341 



€*¥> 
26*09t 
ORGAN ME At J 
3*462A20a^ 



BEST 



copy 



SAS 

TA:iK»ie 

ANALYSIS OF VAHIA^ :C PfJOCEOURE 
CLAdS Ll£VuL INFCRMAtlON 
CLASS LCVELS VAL.UCS 

ORAUe 3 A 0 C 



NUMOLi OF QflSERVATlONS iN tJV CROUP « 1674 

2:J^uH^^T7i;;;r.^•*:J^*^^5S/^SN^2^^i.^'!;"INN^7s"L=^^J.I? ^^'^ '-""^^^ <« v*lucs. Ho.tvtH. only 



OLPENQI^NT VAklAtlUw: lmmak 

SOUf^CL 1>F 

MOOLL 2 

ERRQR' 1934 

CJHKLC7LO TQTAC I 9d6 



SOUHCc 
G>IAOC 



OF 
2 



:>UH OF SUUARCS 

1344.J1J79962 

ANOVA :aS 
23«iajd4:7/0 



SAS 
TASK>IE 

ANALYSIS OF VARIANCE PRQCCDURE 



MCAN SQUARE 
'>*tt3404670 



F VALUt 
13*90 



PR > F 
0.0001 



F VALUE 
13.90 



PR > F 
0.0001 
ROOT NSE 
0.913261S7 



R^SQUARC 
0.017246 



30.6S4^ 
ORGAN MEAN 
2.9792060r> 



BEST copy 



60 



61 



I 



SAS 
TASKS t A 

ANALYSIS OF VAHIANCC Ff<CCtiaUH£ 
CLASS LEVEL INFOHMATIQN 
CLASS Le^VELS VALUES 

GMA96 3 A U C 

NUMU£R Ot UCSCRVATIONS IN UY OHOUP » 1420 

NUTt.S ALL OLPCl^ULNT VAHI AdLCTS ARd CUNSISTUNT WITH RESPECT TQ THE FRtiSLNCE QR AOSEHCE UF MiSStNw VALUES. HUliEVERt ONLY I4ll» 

quuwKvatiun:> in uy okuup can oe uslu in this analysis* 



SAS 
TASK^l A 

analysis of variance PPOCEQURE 



OliPLNOLNT 


VAKl AUi.CS 


Content 












sauRCL 




or 


SU;; OF SQUARES 


MCAN SQUARE 


F VALUE 


PR > F 


R-SOUARE 


MUDEL 




2 




46«833&a458 


43*44 


; ^0 1 


O.0S7922 


CilROR 




1413 


1323*491474/0 


1 •076I68O6 




ROOT MSE 




CORRECTbO 


rOTAC 


1413 


l6i7»Uao4407 






1 •03834672 




SOURCE 




OF 


ANOVA SS 


F VALUE PR > F 












2 


9J»u67lfrVJ7 


43*«4 OtOOOi 









C.V- 
33. 204*. 
CONTENT MCAII 
3«t271tS««« 



I 



62 



lERlC - 



BEST copy 



4h 



6:^ 



SAS r* 

TASK^tU 

ANALYSIS OF VAHIANCC PftUCbOUAC 
CLASS LCvlL INFOHMATION 
CLAJS LEVlLS values 

GRAUC 2 AO 

HUHHOi OF OBSERVATtONS IN 8Y CNtiUP « 1430 

NUTd: Ai.L OdPuHUk;{4r VAKiAiiLb'S ARB CUNSUTtNT wiTH MCSPLCT TU THC PkCSLNCC AUSENCI- UF *1ibSiNG VALUES* HUViiVER* UNLY 1432 
OUscHVAl l(JN:i IN uY QROUt* LAN dtL U:»C£U IN THIS ANALYSIS* 



UEPLNUC^T vahiaull: LUNTCNT 

SUUkCl Of 

MOOEL 1 

EHRvjR 1 #30 

CURc^^CYcU rCTAL 1431 

SOURCL OF 



SUM UF :»UUARCS 
84«264a5719 
Ib96*32bii4a6 

iret>«S94972a7 

ANOVA SS 
a9«2u9(ib7l9 



SAS 
TA:^K>ltt 

ANALYSIS OF VARIANCE PRUCtOURC 



MEAN SOUAHE 
d9«269ab7t9 
1« 18624134 



F VALU£ 
75*25 



PR > F 
0.000 t 



F VALUE 
7S«2b 



PR > F 
0«ti001 
ROUT Mse 
1 •oevi47u7 



2; 



-SOUAME c.V. 
0.04V994 J3«299i 
CONTENT MCAII 
3«2791396<* 



4k 



64 



BEST COPY 



65 



I 



SAS 
TASK'IC 



ANA4.VSIS OF VARiANCC PHOCEOURE 
CLASS LEVEL INFORMAIION 
CLASS LEVCcS VALUES 

UHADE 2 AH 

NUMBLM OF adS^HVATlUNS IH UY GHOWF ■ 139% 



NUT*.: #y.L JLHcNULNT V^KlAJLLb AKC CUNSlivTuNr itflTH RC5PECT TL tHL f'HLbENCi! OR AUSCNCE UF MISSING VALUCb* HOWbVtHi unLY 
Ui^cRVAT luNb IN uY aROUH CAi4 liC U5LU IN THIS ANALYSIS* 



1351 



L 



OCPLNUCNT VAHiAiiuLJ UlNTTNT 

iUURCe UF 

MUO&L 1 

ERROR ~ ' 1 349 

CUKkECTCO total 1 3bU 

SOUHCti OF 

GrAOC 1 



SUM OF SQUARES 
6J«34702S2S 
1363*91514306 
1449«;£62768J2 

ANUVA SS 
63«34r625«:S 



SaS 
TASK«1C 

ANALYSIS OF VARIANCE PROCEDURE 



MEAN SCUARE 
Q3«34762b25 

1*027364 ea 



F VALUE 
61 



PR > F 
0*0001 



F VALUE 
61*66 



PR > F 
0*0001 
ROOT MSC 
1*01359007 



R*SQUARC c*V« 
0*043710 30*30tJ 
CONTENT NeAf« 
3*3449a9«j 



6t) 



BEST copy 



ANALrStS OP VAHlANCC PftUCt^uURC 
CLASS LCVCL INHCHMATION 
CLASS LEVISLS VaLUCS 

CHAOE 3 ^ « C 



NUMdEH OP OGSCMVATlUhS IH tSY ^HUUP • UT4 
2^.^H$!:Ti2Nri.J:'*2^*l!^J;Sp*2i(N''^f i^^S**!./!;,?/'^ PHtSfNCC OH AdSCNCE OF HISSIHC VALUtS. HO.CVCH. ONLY IS8« 



Ol-PCNULNT VAHl AdLi.: CUNT k:NT 

SOURCC Of 

MOOcL 2 

CRROR 1 505 

CUR»4LLTt;U TUTaL 1 A87 

source Of 

GRAUk 2 



3MM OF SUUAftcS 

1449*01 lOttJl^S 
ANUVA ^3 

2d« 174y3di4 



:;as 

ANAL <1 IS UF VARIANCE PRCCEOURC 



MEAN tjyiAHt 
12*56746907 
0»C <J^407 



F VALUE 
14.00 



pn ^ f 
O.OOOt 



^ VALUl 
?4tC0 



r»R > F 
0.0001 
ROCT MSE 
0.94806332 



ft*SOUARE C«V. 
0.0t?364 36.103/ 
CONTENT MCAH 
2.6£594A5 • 



68 



BEST COPY 



SO 

6.9 



ANALYSIS Of VAHlANCC PROCLDUftE 
CLASS LEV6L iNfCRMATlON 
CL%SS LLVELS VALUES 

CMADE 3 ABC 

NU^QCK UIF ueuERVATlUNS IN UY UHUUP « U20 

HUTCJ ^L OtHLNUCNT VANl A0LE5 AME CUnSIUTLNT »lrH MbSPb'CT TO THti PHl^SkNCL UR ABSENCE OT MISSING VALUt£S« HuwLVCR* ONLY t4l6 
OaS4.NVATllJN> IN UY CAN UC U:*i:U IN TmI5 ANAuYSlSt 



OEP£Nt)£NT VARIAIIL.E: aTY(.E 

SOUHCC OP 

MOOcU 2 

CRROR 1413 

COflRI£CTt;0 TQTaL 1415 

Su* . OP 

CRAi>E 2 



SUM OF SOUARLS 
49t J47l0«/i0 
t4 32«262396^a 
t4dl«60^4o32tf 

ANUVA JS 
49«34?10630 



SAS 

Task*! a 

ANALYSIS OF VARIANCE PRQCEOURC 



MEAN SQUAkC 
24«a732tS3t9 
lt01363224 



F VA4.UE 
24«3A 



PR > F 

O.OOOl 



F VALUE 

;S4«J4 



PR > F 
0*0001 

RCOT MSC 
U0067930S 



R*SQUARE 
0*033306 



c«v. 

32*989'^ 
STYLE MEAt/ 
3«0II96«92/ 



BEST copy 



i 



SAS 
TASK«t0 

Analysis of vahIance PwoccouRt 

CLASS LCVCL INFORMATIUN 
CLASS LCVEUS VALUES 

graoc z a a 



NUMbCH OF OdSENVATlONS IN GkOUP a 1436 

NOTC: ALC Ut;P^NDkNT VAki AULtS Mil CUNS1;«TLNT irflTM RCSPECT TO THC PRCStiNCC OR AU^ENCC OF MISSING VALUCS. HQtfirVtfl. ONLY 1^49 
a3;»u«VATlUNi IN UY GHOUP CAN UC USLU IN THIS ANALYSIS. ^"^-^tn^"- ^U^CN^-u ur M19S1NQ VALUtb. HOWLVtR. ONLY I#J2 



SAS 
TA t B 

ANALYSIS OF VARIANCE PHQCEOURE 



OLPtNUdNT 


VAHt au(.l: 
















S( URCC 




OF 


SUM UP SUUAHCS 


MCAH StiUAHE 


F VALUE 


PR > F 


R-SOUARE 


c*v. 


HOOEL 




t 


93»79U70b4S 


SJ. 794170545 


54*99 


0*0001 


0*037030 


29* 939; 


ERROR 




I 430 


1399.0609314? 


0.9783642S 




ROOT HUE 




STYLE MEA.« 


CURRIiCTbO 


TOTAi. 


I 4JI 


I452»fli>9636d7 






0«9evl2299 




3*3037709.. 


SOU MCE 




OF 


ANOVA SS 


F VALUE PR > F 










G»)AOi; 




X 


SJ«79U/0t>49 


54*99 O.OOOI 











I 



72 



BEST COPY 



73 



SAS 

TASK»tC 

ANALYSIS OF VARIANCE Pl^UCEUURE 
CLASS LCVtlL iNFCkNATlQN 
CLAUS LEVCLS VALUES 

UHAUE 2 A U 



NUMtfEH OF OHSEHVATIONS IN Uf <^ROOP m 13^4 



L 



L. 



lERiC 



3AS 
TASK* 1 C 



UEPtiNJEHT 


VARI ^B4.wt 


STTLE 




I SUURCC 




OF 


^UM OF SQUARLS 


MODEL 




t 




' rRROR* 




1 348 


124I«2422«»JJS 


CORrttvCTLO 


TCTAL 


t 349 


IJOdtOtiOUOOOO 


SOURCC 




OF 


ANOVA SS 


<iRADr 




I 





ANALYSIS OF VARIANCE PROCEDURE 



MEAN SQUARE 
64»7U7/3665 
0«920a0287 



F VALUE 
72*90 



PR > F 
0«U001 

ROOT Mse 

0*95958474 



R*SOUARE 
0«091030 



C«V« 
2««7«7t^ 
STVLC nCAfl 
J •3433333.) 



F VALUt 
72«b0 



PR > F 
0«000t 



BESTCG y 



TASK«1E 

ANALYSIS OF VAPIANCC PHOCeOUR^ 
CLASS ^evCL INFORMATIUN 
CLASS LEVELS VALUES 

CHAOe z ABC 

NUrttiUR OF OHbtHVATlONS IN UV ^MQUP m lo74 



OCPOIULNT VAHlAULu: ^rVLE 

souncE OF 

mOtL 2 

ERROR' 1964 

CORHECTCO TOTAu | 5a6 



SUURC& 
GHAi>«£ 



OP 
2 



SUM OF SUUAAl,S 

U62.237I3036 
ttdy«25Sl9649 

ANOVA SS 
27tOi606ttl2 



SAS 

Task* IE 

ANALYSIS OF VARIANCE PROCEOURC 



Nb'AN SQUARE 
t3«S0903406 
0« 73373590 



F VALUE 
iet4l 



PR > F 
0*0001 



F VALUE 
le«4l 



PM > F 
0«000t 
ROOT MSE 
0«69o$0Jo6 



R* SQUARE 
0«0227ie 



2«« 162^ 
STY i: WEAN 
3«04|9II790 



BEST COPY 



77 



SAS 

TASK 6 ANALYSIS OF VARiAHCC PROCEOUPC 
CLASS LEVEL |NPORMAT|0^ 
CLASS LEVFlS values 

GRADE 3 A 8 C 



BEST COPY 



NWMOER OP GUSERVATtONS IN DATA SET • 2472 



note: all 0LMEHDLN7 VAk lABLES ARE CONSISTENT WITH RESPECT TO THE PRESENCE OR ABSENCE OF 

0B:>ERVAT1UNS in data set can OSCD IN THIS ANALYSIS* 
I SAS 



OuPENDklNT VARSAdLt;: UI 

SOURCE OF 

NOOCL 2 

ERROR 2405 

CORR&CTEU TOTAL 2407 



MISSIKKr VALUES* HOllEVERf ONLY 2400 

12:90 VEONfiSOAYc NOVEMBER 27 t I98S 



SUM OF SQUARES 
332* 67751902 
3254«6972Jiei 
J567«S747&0a3 



ANALYSIS OF VARIANCE PROCEDURE 



MEAN SQUARE 
166«43e75y51 
1*35330446 



F VALUE 
122*99 



PR > F 
0*0001 
ROOT MSE 
I*l6331tiI5 



R*SQUARE 
0*092 78b 



C#1 
37»t71 
01 Mfti 
3« l299Mi 



SOURCE 
GRADE 



1 



OF 

2 



ANOVA SS 
332*67751902 



F VALUE 
122*99 



SAS 



PR > F 
0*0001 



ANALYSIS OF VARIANCE PROCEDURE 
CLASS LEVEL INFORMATION 
CLASS LEVELS VALUES 

GRADE . ABC 

NUMBER OF CBSERVATIONS IN DATA SET « 2472 



I 
I 
I 
I 



note: all OEPENUENi VA»ilAeLES ARE CONSISTENT «ITH RESPECT TO THE PRESENCE OR ABSENCE OF MISSING VALUES* HOVEVERt ONLY 235B 

0BSC:RVAT£0NS IN OATA SET CAN BE USEO IN THIS ANALYSIS* 
I SAS 



DEPENDENT VM4lAaLE: COMP 

SOURCE OF 

MOOtL 2 

ERMOH 2355 

CORRCCTHO TOTAL 2357 



SOURCt* 



1 



OF 
2 



ANALYSIS OF VARIANCE PROCEDURE 



SUM OF SQUARES 
41&*6ai25499 
1932*61462669 
2346*69506163 

ANOVA SS 
4l9*86l2tl499 



MEAN SQUARE 
207*94062749 
0*62072606 



F VALUE 
293*36 



F VALUE 
293*36 



9AS 



PR > F 
0*0001 



PR > F 
0*0001 
ROOT MSE 
0*90594044 



R*SOUAR£ 
0*177069 



29#9B' 

COMP MC^ 
3*0690t6( 



79 2 



BESV COPY 



L. 



ANALYSIS OF VARIANCE PROCEOUHC 
CLASS LEVEL INFORMATION 
CLA.SS LLVCLS VALUES 

CMACE 3 A d C 

NUMbLR OF OBSLRVATIONS |N DATA SET ^ 2472 



€-_Jti_«;^lwr;_yALXij^*U-Mi)*f-vrR« only 5Ja<# 



SOURCt£ 
GRADL 



OF 
Z 



ANUVA SS 



F VALU^ 
2£3t36 



9AS 



PR > F 



ANALYSIS OF VARlAhCe PROCEOURE 

CLASS LEVEL INFORMATION 
CLASS L*,VCLS VALUCS 

^'ACE J ABC 

NUMBER OF OBSERVATIONS IN DATA SET « 2472 

""'"^ 5^'sL^^:^ys^r,rsi?2i^*LVB"rii;!riN*Tms"sssfrs,;° °'' "^"•^'^ =»' 

^ SAS 12190 HEONESOAYf NOVEMBER 27 • 1969 

ANALYSIS OF. VARIANCE PROCEDURE 



OCPCNULNT v^iaule: ^TYLC 



L- 



SOUHLli 

MOlJEL 

LRRUR 

CUHNCCTEO TUTAL 

SOURCE 
GHADC 



t 



OF 
2 

2J56 
2396 

OF, 



:>UM OF SQUARES 
276*22610469 
1 977«0606B132 
22S9«2d69a60t 

ANOVA SS 
276* 2261 0419 



MEAN SQUARE 
139*1 1309235 
0«e39159€7 



F VALUE 
I69«7a 



PR > F 
0*0001 
ROOT MSE 
0*9169Sh7S 



R-StUAr^E 
0* 12' 306 



£6«63« 

STVLC ME A 
3# I 767691 



F VALUe 
169*76 



PR > F 
0*0001 



SAS 

ANALYSIS OF VARIANCE f^POCEOURE 
CLAS'S LEVEL INFORMATION 
CLASS LEVEL9 VALUE5 
GRADE _ 3 ABC 

NUMBER OF OBSERVATIONS IN OATA Sry m 247Z 



. . . ._ . _SAS 

O - - . 

8G 81 



I 

I 
I 

I ' 

! 

I 
I 

i 

» 

I 

I 

I 

I 

I 

i* 

0 



ANALYSIS U* VARIANCE PROCEDURE 





VAMiAUwES 


JHit AN 










iOUKCE 




DF 


SUM Of 50UAMES 


MEAN SQUARE 


r VAl.UE 


PR > F 




• 


.2 




♦••^ •78^37422 


232*12 


0*0001 


coRi^tcri-u 


TOTAL 


C *l 9 

2397 


2iJlS«2O6lO60> 


.as!67ea 




ROOT MSE 
0*99920366 


sauHi.1: 




OF 


ANUVA SS 


^ VALUE PR > f 






1 




2 


46J*87874a4^ 


^32.12 O.OOOl 
SAi 







R*SQ'JARE 
0*164670 



34*00t 
OROAN M$ 
2*9399913 



ANALYSIS OP VARIANCE PROCEOiIRE 
CLASS LEV'iL INFORMATION 
CLASI LEVLLS VALUES 

GRADE 3 A 6 C 



NJHdER :iSCRV/»TION, IN DATA SET « 

HOTg: ALL_tcPt:NUENT .VA# MLC% AKd. CU ,SISTfcNT^«ITK. ?5I^ r-XT.Tn _TMf?^Pl?P«irMr F . 
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The significant differences for grade level support the sensitivity of 
the scoring rubrics to instructional exposure. Because the data will be 
reanalyzed by classroom level, distributed into strata by geography, etc., 
no Inferences are made at this point. Preli^ nary correlational analyses 
were conducted to ascertain the relationship among scales. On Inspection, 
these vary between .6 and .9. Our National Repurt will describe these 
analyses in detail. We also anticipate reporting the double scored 
reliabilities and the adjusted scores based upon calibration essay scoring 
for the raters. In addition, tasks 3, 5, and 9 will be analyzed. 
Next Steps 

Our next steps depend upon additional resources. We hope to secure 
support th'^ough OERI for the following efforts: 

1. To create data files for student questionna *e and attitude data. 

2. To create files for teacher and school level questionnaires. 

3. To conduct descriptive analyses. 

4. To conduct relational analyses based upon data In school and 
teacher questionnaires in order to identify promising 
instructional practices. 

5. To prepare school level reports for local schools ana states. 

6. To prepare a report of the U.S. National Study. 

7* To prepare files for international analysis of the USA data. 

Were additional significant resources available, we would ideally like 
to score the remaining data from each classroom. We would also like to 
prepare qualitative analyses of the student papers, focusing on what they 
?»aid, rather than mode of expression. 
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Summary 

Our study has been successful to this point. We have achieved an 
appropriate response level and believe we have provided technically 
competent work within both time and budget constraints. We plan to share 
the results of our future analyses (wfen we have developed the full 
analyses) with researchers and practitioners to develop sound i:*ferences 
about the quality of writing of American students. 
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SUPPORTING DOCUMENTS* 



★ 



Due to the cost of duplication, only one copy of supporting 
documentation -^s provided. 
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TASK 1 TRAINING AND SCORING MATERIALS 
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SCORING GUIDES 



Introduction ; You are to score a number of compositions vritten by a wide 
range of students from your country and to rate these compositions according 
to a scoring guide and a set of compositions. whose scores have been determined.. 
The guide and the set of compositions represent the consensus of experts from 
over fifteen countries as to the characteristics of a good composition on this 
task. You have probably taught a number of students like the ones whose 
compositions you will be scoring. In this case, however, you do not know the 
student personally and you have only these examples of their writing on which 
to Judge the student's performance. As you score each composition, you should 
try to consider it in the light of the scores given the set of compositions 
that your group leader has provided ; they will form a benchmark reference Tor your 
scores . It may be that you will not find a composition THaFTiaches the peak of 
perfection that you would hope for from an experienced published writer; these 
are in most cases first drafts and you are asked to judge these compositions ac- 
cording to a consensus as. to how well a student of this age or level of education 
can write In a limited period of time on a topic that is relatively unfamiliar to 
that student. 

How you will score ; A. You are first asked to rate each composition 
according to your overall impression of the quality of the composition. Your 
overall impression is usually based on your sense of how well the composition 
fulfills the task and topic requirements and your sense of its quality as a piece 
of writing relative to other students in the age-group that completed that task. 

B. In addition you are asked to rate each composition as to its quality 
according to a number of specific dimensions: its content, its organization, 
its style and tone, its uses of grammar, orthographic conventions (i.e., spelling, 
punctuation), its neatness, and your personal reaction to the composition and its 
writer. 

To help you make these Judgements, we have had a set of compositions rated 
by a Jury of er.perts in composition from several countries. They have determined 
that certain aspects of each composition need to be rated separately although 
they agree thai an overall impression is also Important. 

The Jury has developed a scoring scheme which you are asked to use, and , 
they have created international guidelines and definitions of certain aspects 
within that scheme. The Jury has also selected the benchmark compositions to 
Illustrate various levels o-^ performance by students with respect to the dif- 
ferent parts of the scheme. Some of the :ompos1t1ons were written by students 
in other countries who were asked to write on the same topic. You are, there- 
fore, asked to judge the compositions you will be reading according to a n 
International scale . 

You are going to spend several hours practicing using that scale. This 
guide is an introduction to the scale and the scheme of .irking. 

The Score Sheet . Fl each type of coitiposltion you will be given a score sheet. 
The sheet is designed so that you can Indicate your ratings of ten coTiPCSitions, 
A sample score sheet for Task Five appears on the next page. You must first 
enter '^he Student Number. Check to make sure you have the Student Number 
exactly. The Task Number has already been printed on the second row of the sheet. 
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SmLS J°f^!^?J ?^ " *° t^'e specific topic. Enter the Topic 

Number below the Task Number If you can determine which topic (1-7) the student 
selected. Then you must enter the number you have been assigned in each space 
on the row marked Rater Number, If you find that you cannot score the paper 
Decause the student wrote nothing or because the paper Is Illegible or ^or some 
other reason, indicate In the box In row 5/6 the reason for not scorlnr the 
composition using one of the codes 7-9. but DO NOT enter any scores beyond 
this point. If you do score the composition.* Indicate your Overall Impression 
In the box In row 5/6 and proceed down the column, entering a score for each 
aspect to be scored. DO NOT OMIT A SCORE. 

.Overall Impression. Read the composition In a normcil fashion, without stopping 
to mark or underline segments of the composition or to write comments. Record 
your first impression of the over quality of the composition. Indicate on 

I I V 5 ^^^^ 'l*'''^^ ^'^^^ think cf the composition 

as a whole. DO NOT CHANGE THIS SCORE ONCE YOU HAVE RECORDED IT. 

Rate the composition according to your overall impression of its merit. DO 
NOT attempt to rate your estimate of where it would fall on a normal cjrve of 
pv-rformance. but your sense of its adequacy or excellence as a composition 
according to the benchmark scores. Thus, you should not award a certain pro- 
portion of low. intermediate, or high grades, if youBo~not perceive such a dis- 
tribution In the set of compositions you are asked to score. 

Detailed Impression . After you give your overall impression of the composition, 
you will also rate each composition on a number of detailed dimensions, which 
are slightly modified or expanded according to the nature of the writing task. 
These dimensions represent an international consensus of teachers and judges of 
writing about those aspects of a composition that should be singled out for 
attention. Somt dimensions may be more important in one country; other dimen- 
sions in another. We are seeking to have each country's students judged according 
Jo an international set of standards, so ycu should consider each dimension equally. 
You do not have to try to make your rating of each dimension match o« average to 
equal your overall impression. 

AFTER YOU RATE ALL THE DIMENSIONS. DO NOT CHANGE YOUR OVERALL IMPRESSION 

SCORE. 

The dimensions that will be incluc'sd are: 

QUALITY AND SCOPE OF IDEAS. A dimension that focuses on your impression 
of the content of the composition, what it says and how '"ully and completely it 
says it. 

DRRANIZATION AND PRESENTATION OF CONTENT. How the student arranges trie 
material and the structure cf both the composition as a whole and the individual 
segments or paragraphs. 

STYLE AND TONE. Including the choice of words and phrases, sentence 
structures and larger units of discourse, the variety and flow of sentences, 
and ;'Our sense of whether the student has made an effective nd appropriate 
use of the language given the aim. context, and audience of the composition 
task. 
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SAMPLE SCORING SHEET TASK 5 



1. Student Number 






















2. Task Numbar 
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15 


15 
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3, Topic Number (1-7) 






















H. Rater Number lyour nunDer; 






















5, Composition Cannot Be scored \i^vr 






















6. Overall Impression u-bj 






















7. Quality and Scope U-b; 
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llf Overall structure vl-o; 






















12. control OT uetai 1 vi-b; 






















1j. style and lone vi*b; 






















I'f , Lfioice and LonsiStency or lone va-o; 






















ID. unulCc CT WOruS aHu rnraScS ^1*D; 






















lO. uramiTlatlCal reatUreS \1-D; 






















it 9 opciiing ana urtnugrapny 






















lo. nanawriwiny Vl'*t>/ 
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20. Sense of Connection with Writer (1-5) 























*If the composition cannot be scored, m^rk in row 5 one of the following: 7 

8 
9 



off topic 
illegible 

blank paper, no response 
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LEXICAL AND GRAWIATICAL FEATURES. Including both lexical and syntactic 
features to Indicate your impression of vhe student's mastery of these features. 

SPELLING AND ORTHOGRAPHIC CONVENTIONS. Including those conventional 
aspects of punctuation to indicate your impression of the student's mastery 
of these features. 

HANDWRITING. To indii ^ your sense of the student's physical presentation 
of the composition including iegibility and clarity of corrections and paragraph 
signals. You should remember tHat a student may not have had time to recopy 
the whole composition. 

RESPONSE OF THE RATER. To provide you with an opportunity to indicate 
your interest in the composition, and your intuitive sense of the writer as 
seen through the composition, whether you are attracted to the person or are 
persuaded by what the person has written. 

You should rate each composition following the order of these dimensions 
that have been outlined on the scoring sheet. MARK EACH COMPOSITION ON EACH 
SCORE CALLED FOR. 

Sub-category Ratings . On Tasks 5, 6 and 7, which are more complex, you 
will be asked to rate the composition on certain particular aspects )f one or 
more of the dimensions. Arain these are asperts that the international jury 
has agreed should be rated independently. You should rate the category itself 
first and then the sub-categories. The category score is not an average of 
the sub-categories for it includes more than their sum. There vill be detailed 
explanations of these aspects in the scoring guide for the particular task. 

B enchmark Compositions . Attached to the scoring guide fo each task is 
a set 0^ compositions that have been selected by the internationr jury to help 
you understand the interpretation of the scoring guide and its application to 
specific compositions. As you read through the scoring guide, you shoul'^ also 
read through the benchmark compositions, and. the associated commentaries. Doing 
so will help you become familiar both with the dimensions that will be rated and 
with the international standards that set the scale from Excellent to Inadequate 
on each of the categories and sub-categories. While scoring compositions after 
the training session, you should frequently refer to the benchmark compositions 
to help you to apply the international standard. 
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TASK lA: Description of a Bicycle 



A. QUALITY AND SCOPE OF CONTENT, In making this judgement, you should look 
for: 

1. whether the student has provided sufficient characteristics Ce.g.. 
color, type — racing, touring, boy's or twirl's — and accessories) 
so that the uncle could purchase the. bicycle requested, and; 

2. whether the student has omitted superfluous details (e.g., explaining 
that a wheel has spokes). (In some cases the student has described a 
bicycle other than those pictured. Unless forbidden in a country's 
directions to the student, this response is acceptable.) 

8. ORGANIZATION AND. PRESENTATION OF CONTENT. In making this judgement, you 
should determine whether the characteristics are grouped naturally into a 
structure that makes sense to a reader seeking to Identify a bicycle. The 
writer should not make many false starts. 

C. STYLE AND TONE. In making this judgement, you should consider: 

1. the selection of appropriate terms to describe the features of the 
bicycle. 

2. the extent to which the specific words, phrases, and larger units of 
discourse show that the writer is aware that the audience of the letter 
is a generous older person. You should be aware that the stimulus con- 
tains the opening sentence so that a good letter may appear to begl.j and 
end abruptly. 

D. LFXICAL AND GRAMMATICAL FEATURES. To be supplied by National Centers. 

E. SPELLING AND ORTHOGRAPHIC CONVENTIONS. To be supplied by National Centers. 

F. HANDWRITING AND NEATNESS. To be supplied by National Centers. 

6. RESPONSE OF RATER. You should give an estimate of your Interest in the 
composition and/or the writer. 
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TASK IB: Self DescKptlon 



A. QUALITY AND SCOPE OF CONTENT. In making this judgement you should con- 
sider the use of essential and relevant information includina sufficient 
observable characteristics, and distinctive characteristics concerning 
clothes, the place where the writer will be, or other information appro- 
priate to the situation. 

B. ORGANIZATION AND PRESENTATION OF CONTENT., In making this judgement you 
should consider: 

1. the use of continuous discourse (not a list of features); 

2. the orderly presentation of information to provide a clear description. 

C. STYLE AND TONE. In making this judgement you should consider: 

1. the selection of descriptive terms. 

2. the appropriate choice of words, phrases and larger units of discourse 
to indicate the writer's awareness of the individual who is the audience 
(the writer has the option to be casual and informal or relatively formal). 

D. LEXICAL AND GRAMMATICAL FEATURES. To be supplied by National Centers. 

E. SPELLING AND ORTHOGRAPHIC CONVENTIONS. To be supplied by National Centers. 

F. HANDWRITING AND NEATNESS. To be supplied by National Centers. 

G. RESPONSE OF RATER. You should give an estimate of your interest in the 
composition and/or the writer. 
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TASK IC: Letter to Headmaster 



A. QUALITY AND SCOPE OF CONTENT. In making this judgement, you should 
consider that there are 5 relevant facts we might expect in this message: 
1) mentioning of appointment, 2) mentioning of fact that the writer is 
hindered from coming, 3) mentioning of reason tor non-appearance, 4) apology, 
5) name of student. 

B. ORGANIZATION AND PRESENTATION OF CONTENT. In making this Judgem nt, you 
should consider that there is not one single order of giving the four 
elements mentioned under A* You should decide If the content is pre- 
sented in an orderly manner. 

C. STYLE AM) TONE. In making this judgement, you should consider: 

1. the tone of the message should be in accordance with the relationship 
betv/een headteacher (or principal) and student; 

2. the style should show an appropriate use of words, phrases, and larger 
units of discourse. 

D. LEXICAL AND GRAMMATICAL FEATURES. To be supplied by National Centers. 

E. SPELLING AND ORTHOGRAPHIC CONVENTIONS. To be supplied by National Centers. 

F. HANDWRITING AND NEATNESS. To be supplied by National Centers. 

G. RESPONSE OF RATER. You should give an estimate of your interest in the 
composition and/or the writer 
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TASK ID (Population A); Informal Note to Family 

^' fuil^lh^f\f?^^ °[ f^^^^l!^- ^" "^•^^"9 Judgement, you should con- 
?hp Li^Lc*^'^ °^ essential and relevant information includes 

inteJ w?H tll'll' ?^ "^tl' '^f ^''•^'^^ Whom the 

L-c/h ^"'^ ^S'^ '^^^^^^ will remain. The writer should 

or lJJlllf''lT- " bare listing of information and a more el a b- 

or^. presentation are acceptable. 

^' ?Eo'^!'d^IL^^^''^h''^^^^^'^"°I! O^COJTENT.* in making this Judgement, you 
should consider the clear and orderly presentation of information The 

Z'Vyjll TJ 5^°'*' infonnJtion must be presISted naJ 'order 

so that the reader can gather sufficient detail. 

^' «riL5c° ^" ^^^^ Judgement, you should consider the choice 

!Ih?c 55""^' 5"'' ^^''S^'' ""^^^ °^ discourse including the use of 
TtlLot other words appropriate for an info^pial message to 

a member of one's own family. ^ 

D. LEXICAL AND GRAMMATICAL FEATURES. To be supplied by National Centers. 

E. SPELLING AND ORTHOGRAPHIC CONVENTIONS. To be supplied by National Centers. 

F. HANDWRITING AND NEATNESS. To be supplied by National Centers. 

G. RESPONSE OF RATEP. You should give an estimate of your interest in the 
composition and/or the writer. 
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TASK IE (Population B and C): Application for a Holiday Job 

A. QUALITY AND SCOPE OF CONTENT. In making this judgement, you should 
consider: 

1. whether the letter clearly refers to the job for which the candidate 
is applying; 

2. specifies the period when the candidate will be available; 

3. Indicates that the applicant is qualified for the position by aoe or 
educational level ; 

4. gives the rame and address and/or phone number (although in somii countries 
that information can be on the envelope, it might well be referred to in 
the text of the letter). 

B. ORGANIZATION AND PRESENTATION OF CONTENT. In making this judgement, you 
should consider: ^ 

1. the clear and orderly presentation of information including the use 
of appropriate format for a letter of application; 

2. the orderly presentation of information called for in the advertisement. 

C. STYLE AND TONE. In making this judgement, you should consider that the 
choice of words, phrases, and larger units of discourse (such as modes of 
address and salutation) are appropriate to a formal letter of inquiry. You 
may wish to assume the role of the recipient and decide whether you would 
be favorably inclined towards the writer. 

D. LEXICAL AND GRAMMATICAL FEATURES. To be supplied by National Centers. 

E. SPELLING AND ORTHOGRAPHIC CONVENTIONS. To be supplied by National Centers. 

F. HANDWRITING AND NEATNESS. Tc be supplied by National Centers. 

li. RESPONSE OF RATER. You should give an estimate of your interest in the 
composition and/or the writer, including the degree to which you might 
respond favorably to the writer. 
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Explicit Features Used In Task 1 Training and Scoring 



TASK lA 


CONTENT 


ORGANIZATION 


1 STYLE TONE 




Unique characteristics, type, 
boys/girls, 10 speed?, handle- 
bars, baskets, tralningwheels, 
horns, pockets, lights? 






5 


All Items i cmmnts or func- 
tion, desirability & recogni- 
tion of reader 


Must be cohesive & coherent 
,has Intro, body, & conclusion 
paraphrasing, follow a progres- 
sion a plan tn description eg, 
top to DOttom or general to 
specific; no digression or 
lapses 1 • *!og1c 


Effective use of descrip- 
tions, sentence variety, 
acknowledging reader 
politely, but no obsequi- 
ously, shows gratitude 


4 


All Items there but not all 
the detail 


Includes all In 5 but plan not 

& coherence - does not require 
paragraphing 


All of 5 but used inconsis- 
tcntiy 


3 


Missing detail, but not enough 
to lose Identification of the 
bicycle - some superflures de- 
tail ok 


No dill berate plan, however .. 
organization does not Impede 
comprehension 


Lack of descriptors, and 
.sentence variety - little 

acknowledgement of reader 
wedk diction 


2 


Can't tell I.D. of bike, 
vague 


Confusion, no discernable plan 


Somewhat like a list, not 
obv1ous,pol1te, or grateful 


1 


Can't tell I.D. of bike - have 
not fulfilled task, missing 
Information 


Incomprehensible 


Rude, 1e demanding, no 
acknowledgement of reader 



10 



I 

f 

I 

I 



TASK IB 


CONTENT 


ORGANIZATION 


STYLE'tONE 




Observable distinctive charac- 
terlstrlcs, such as physical 
characteristics, size, shape, 
hair, coloring, age, sex, 
clothing and ether people or 
props; may mention a special 
or specific meeting place 




f 


5 

i 
i 


Can easily Identify person 
Includes several distinctive 
characteristics 


'Must be cohesive & coherent 
has Intro, body, S conclusion 
paraphrasing, follow a progres- 
sion a plan in descriDtfon ea. 
top to bottom or general to 
specific; no digression or 
lapses in logic 


Effective use of descrip- 
tions, sentence variety, 
acknowledging reader warmly | 

III ir (ciiuij iiijinncrt tiut no 1 

obsequiously, shows grati- 
tude 


4 


Can Identify, has some distin- 
ctive characteristics 


Includes all in 5 but pUn not' 
as clear - must have cohesion 
& coherence - does not require 
paragraphing 


Al 7 of 5 but* iicoH 4nmne4#Ml 

nil VI V wu V uscvi incun5i5 j 
tently 


3 


Probably can but might have 
some difficulty; too many com- 
mon characteristics but not 
enougn unicjue 


4 di liberate plan, however 
organization doss not impede 
comprehension 


Lack of descriptors, and. 1 
sentence variety - little 1 
acknowledgement of reader ( 
weak diction 


2 


Can't I.D, the person;1ncludes 
some Irrelevant features 


Confusion, no discernable plan 


Somewhat like a list 


1 


Includes mostly Irrelevant, 
unobservable features; can't 
Idei.tlfy person 


Incomprehensible 


Rude, no acknowledgement 
of the reader 
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CONTENT 


ORGANIZATION 


STYLE TONE 




Reason for nonappearance, 
apology, name^ consideration 
for principal 's time 






5 


All of the above items with 
some details, explanation and 
perhapr alternative suggestion 


correct fonnat: greeting, body, 
closini,; cohesive; "paragraph- 
ing" when appropriate; no 
digression 


For^'il, courteous^ recogni- 
tion- authority of prina'pal 

meSSaoe sounds slnrora anM 

the reason is important, 
choice of vocabulary, sent- 
ence variety 


4 


All items there but not as 
elaborate 


correct format, minor problems 
in cohesion, a small digression 


All of 5 but loee vaf1tt*\i 

In sentence structure 


3 


Missing one key item (bare 
bones) adequate 

1 


Correct format, very short 
eg: 1-2 short sentences, does 
not Impede reader understanding 


little acknowledgement of - j 
reader - lacks sentence •! 
variety 


! 

2 


Missfng t/o or more 


flawed format, impedes reader ' 
unde'^nanding 


Somewhat list-like not ob- 
viously polite poor diction 


i 


Declaration of absence or ex- 
cuse 


No format, confusing, digres- 
sion 


No a cknowledgement of • 
reader, rude, telegi aphic 
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TASK ID 


CONTENT 


ORGANIZATION 


style/tone I 




Addressee, where writer ts, 
with whom, how long grd., when 
win return, some Indication 
of who writer Is 


.."P.S," - If used, P.S. should 
not have essential Information 
for task. 

4 




5 


All of above, plus elaboration 
of some, such as show of 
concern, phone #, chaperones, 
safety 


Has sufficient content co have 
Introduction, body ?nc conclu- 
sion, coherence and cohesion 
effecclve 


Shows t*f\T\r(^TT\ ^nr 

I.e., personal Information; 
established tone of safety 
and well-belny; sentence 
variety 


4 


Contains everything but not 
as elaborate or detailed 


Has evidence* of Introductl-jn, 
body and conclusion, but not 
necessarily ell threej coher- 
ence and cohesion effective 


All of 5 but used 
Inconsistently 


3 


Missing one element, 
"bare bones" 


Adequate, "bare bones," but 
too short; minor problems In 
cohesion 


Neutral In tone, lacks 
feeling or concern; lack 
of sentence variety 


2 


Missing two or more elements; 
lack of concern for reader 


Real problem in coherence or 
cohesion; Impedes reader's 
understanding 


Little or no attMimt At 
sentence variety; shows 
little or no concern for 
parent; somewhat like a 
list; poor diction 


1 


Declaration of absence; 
no concern for reader 


No format; no evidence of a 
plan or organization; no 
logical relationships 


Abrupt 



III 



TASK IE 
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CONTENT 



^^^^ ' 

ir»s all but (loe7not el»t>or- 
ate. or less elaborately 
done 



Missing rost elements 



ORGANIZATION 



style/tone 



Hu? l^TbUJ ^ 

phone nor pro 

cohesion, e.9«» 
digression 

reader's understanding 

Tattle or no evidence of 
Vo"ltiIlss1ngt«oo;™re 

No fonwf, confusing; 
digressions 



nizes status of reader, 
isiness-li.e sentence 

variety and. diction, 
iriter sells himself to 
reader 

Not as successTul in 
selling himself - not as 
business-lil^e 



Little sentence variei. * 
inconsistent in tone 



No sentence variety', 
too casual 



NO attempt to vary stri 
cture; rude; demanding 



BEST COPY 
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FULL ROTATION OF TASKS FOR GRADES 6, 10, AND 12 



ERIC 



114 



